Analyzing a practitioner perspective on relevance of published empirical research in Requirements Engineering

Background: Relevance to industry and scientific rigor have long been an area of friction in IS research. However little work has been done on how to evaluate IS research relevance. Kitchenham et al [13] proposed one of the few relevance evaluating instruments in literature, later revised by Daneva et al [7]. Aim: To analyze the practitioner/consultant perspective checklist for relevance in order to evaluate its comprehensibility and applicability from the point of view of the practitioner/consultant in the context of an advanced university classroom. Method: Five master level students in the field of IS assessed a set of 24 papers using the relevance checklist. For each question in the checklist, inter-rater agreement has been calculated and the reasoning that the practitioners applied has been reconstructed from comments. Results: Inter-rater agreement only showed to be slight for three questions and poor for all other questions. Analysis of comments provided by the practitioners showed only two questions that were interpreted in the same way by all practitioners. These two questions showed significantly higher inter-rater agreement than other questions. Conclusions: The generally low inter-rater agreement could be explained as an indication that the checklist is in its current form not appropriate for measuring industry relevance of IS research. The different interpretations found for the checklist questions provide useful insight for reformulation of questions. Reformulations are proposed for some questions.